Methods and systems for image processing with multiple image sources

ABSTRACT

Various methods and systems are provided for image processing for multiple cameras. In one embodiment, a method comprises acquiring image frames with a plurality of image frame sources configured with different acquisition settings, processing the image frames based on the different acquisition settings to generate at least one final image frame, and outputting the at least one final image frame. In this way, information from different image frame sources such as cameras may be leveraged to achieve increased frame rates with improved image quality and a desired motion appearance.

FIELD

Embodiments of the subject matter disclosed herein relate to imageprocessing for systems with multiple image sources.

BACKGROUND

As camera technology evolves from photographic film to digital imagesensors, the size of a camera has decreased to the point that camerasare integrated into mobile devices, such as smart phones, that can beconveniently carried by users. Further advances in optical technologyand image sensor technology has enabled such devices to include aplurality of cameras configured with different focal lengths and fieldsof view, which allow users to not only capture high-definitionphotographs but also high-definition video while easily switchingbetween lenses without manually switching the lens.

BRIEF DESCRIPTION

In one embodiment, a method comprises acquiring image frames with aplurality of image frame sources configured with different acquisitionsettings, processing the image frames based on the different acquisitionsettings to generate at least one final image frame, and outputting theat least one final image frame. In this way, information from differentimage frame sources such as cameras may be leveraged to achieveincreased frame rates with improved image quality and a desired motionappearance.

It should be understood that the brief description above is provided tointroduce in simplified form a selection of concepts that are furtherdescribed in the detailed description. It is not meant to identify keyor essential features of the claimed subject matter, the scope of whichis defined uniquely by the claims that follow the detailed description.Furthermore, the claimed subject matter is not limited toimplementations that solve any disadvantages noted above or in any partof this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be better understood from reading thefollowing description of non-limiting embodiments, with reference to theattached drawings, wherein below:

FIG. 1 shows a high-level block diagram illustrating an example imageprocessing system for processing output from a plurality of image framesources according to an embodiment;

FIG. 2 shows a high-level flow chart illustrating an example method forimage processing image frames from a plurality of image frame sourcesaccording to an embodiment;

FIG. 3 shows a high-level flow chart illustrating an example method forimage processing image frames from two image frame sources withdifferent acquisition parameters according to an embodiment;

FIG. 4 shows a high-level block diagram illustrating an example methodfor leveraging motion estimation from high-frame-rate image frames toincrease frame rate for high-resolution image frames according to anembodiment;

FIG. 5 shows a graph illustrating example acquisition timings of imageframes with different image frame sources according to an embodiment;

FIG. 6 shows a high-level block diagram illustrating an example methodfor image processing image frames from two image frame sources accordingto an embodiment;

FIG. 7 shows a high-level flow chart illustrating an example method forgenerating a high-dynamic-range image based on image frames acquiredwith multiple image frame sources according to an embodiment;

FIG. 8 shows a high-level block diagram illustrating a method forleveraging image frames with a same exposure to improve interpolation ofimage frames with different exposures to generate a high-dynamic-rangeimage according to an embodiment;

FIG. 9 shows a high-level flow chart illustrating an example method forgenerating high-frame-rate video from output of a plurality of imageframe sources acquiring at low frame rates according to an embodiment;

FIG. 10 shows a high-level diagram illustrating acquisition times for aplurality of image frame sources to generate high-frame-rate videoaccording to an embodiment;

FIG. 11 shows a high-level block diagram illustrating an example dataflow for managing multiple image frame streams with adjustments fortemporal coherence and motion blur according to an embodiment;

FIG. 12 shows a high-level block diagram illustrating an example dataflow for managing multiple image frame streams according to anembodiment; and

FIG. 13 shows a high-level diagram illustrating motion vectors for aplurality of image frame sources according to an embodiment.

DETAILED DESCRIPTION

The following description relates to various embodiments of imageprocessing. In particular, systems and methods are provided for imageprocessing for multi-camera systems. An image processing system, such asthe system depicted in FIG. 1, may include a plurality of cameras orimage frame sources that record image frames. The output of the multiplecameras may be processed to produce video with a high frame rate (e.g.,a frame rate higher than 24 frames per second) with a desired motionappearance. To that end, a method for image processing, such as themethod depicted in FIG. 2, leverages the information from the imageframes acquired with different cameras to generate an improved imageframe or video comprising a series of image frames. For example, asshown in FIGS. 3-6, two cameras may acquire video at different framerates and different resolutions, and the motion estimated in the higherframe rate video may be used to improve interpolation of higherresolution, lower frame rate video to produce high-resolution,high-frame rate video. As another example, a first camera may acquireimage frames with different exposure settings while a second cameraacquires image frames with consistent exposure settings, and the imageframes from the second camera may be used to improve interpolation ofthe image frames from the first camera to produce high-dynamic-range(HDR) images or video, as shown in FIGS. 7 and 8. As yet anotherexample, image frames may be acquired at a same frame rate with a largerplurality of cameras and may be combined or fused into a single videowith a higher frame rate, as shown in FIG. 9. By adjusting the intervalsbetween initial acquisition times for the different cameras, as shown inFIG. 10, the motion appearance of the final video may be smoothed orotherwise adjusted as desired. The various image frame streams may beused to adjust temporal coherence and motion blur for a single imageframe stream formed from the various image frame streams, as depicted inFIG. 11, or may simply be combined into a single image frame streamwithout down-stream corrections, as depicted in FIG. 12. Motion vectorsmay be calculated between adjacent image frames from each image framesource as well as between image frame sources, as depicted in FIG. 13.

FIG. 1 shows a high-level block diagram illustrating an example imageprocessing system 100 for processing output from a plurality of imageframe sources 110. The components of the image processing system 100 maybe combined in a shared enclosure (not shown), in some examples, such aswithin a single camera apparatus or a mobile phone. Alternatively, oneor more components of the image processing system 100 may be configuredas peripheral devices that may be communicatively coupled to form theimage processing system 100.

The plurality of image frame sources 110 includes at least a first imageframe source 112 and a second image frame source 114. Each image framesource of the plurality of image frame sources 110 may comprise an imagesensor or camera, for example, configured to acquire one or more imageframes. The plurality of image frame sources may be positioned adjacentto each other, with a known relative positioning. The image framesources may be different types of cameras and/or may be configured withdifferent lenses, for example.

The image processing system 100 comprises an image processor 120communicatively coupled to the plurality of image frame sources 110 andconfigured to receive image frames from the plurality of image framesources 110. In some examples, the image processor 120 may be configuredto control one or more of the image frame sources 110 to adjust one ormore acquisition parameters. The image processor 120 is configured toprocess the image frames received from the plurality of image framesources 110, as described further herein with regard to FIGS. 2-12. Forexample, the image processor 120 may leverage the information from imageframes acquired by the image frame source 112 to improve image framesacquired by the second image frame source 114.

The image processor 120 may comprise one or more physical devicesconfigured to execute one or more instructions. For example, the imageprocessor 120 may comprise one or more processors that are configured toexecute software instructions. Additionally or alternatively, the imageprocessor 120 may comprise one or more hardware or firmware logicmachines configured to execute hardware or firmware instructions. Theimage processor 120 may be single or multi-core, and the programsexecuted thereon may be configured for parallel or distributedprocessing.

The imaging processing system 100 further includes a memory 124comprising non-transitory and transitory memory for storing instructionsto be executed by the image processor 120 to perform the methodsdescribed further herein. The memory 124 may further provide a bufferand/or cache for storing data relating to the processing of image framesfrom the plurality of image frame sources 110, in order to support thefunctions of the image processor 120.

The image processing system 100 further comprises a display 130communicatively coupled to the image processor 120 and configured todisplay one or more image frames output by the image processor 120 tothe display 130. That is, the image processor 120 may present a visualrepresentation of data held or output by the image processor 120 and/orthe memory 124. The display 130 may comprise a display device utilizingvirtually any type of technology.

FIG. 2 shows a high-level flow chart illustrating an example method 200for image processing image frames from a plurality of image framesources according to an embodiment. In particular, method 200 relates toacquiring image frames with multiple image frame sources, and leveragingthe information of image frames from one source to improve the imageframes of a second source. Method 200 is described with regard to thesystems and components of FIG. 1, though it should be appreciated thatthe method 200 may be implemented with other systems and componentswithout departing from the scope of the present disclosure. Method 200may be implemented as instructions in memory 124, for example,executable by the image processor 120 to perform the actions describedherein.

Method 200 begins at 205. At 205, method 200 acquires image frames via aplurality of image frame sources with different acquisition settings.For example, method 200 may acquire image frames with at least two imageframe sources, wherein the two image frame sources are configured withdifferent acquisition settings. For example, as described further hereinwith regard to FIGS. 3-6, a first image frame source may acquire imageframes at a first frame rate and a first image resolution, while asecond image frame source may acquire image frames at a second framerate and a second image resolution, wherein the first frame rate ishigher than the second frame rate and the first image resolution islower than the second image resolution. As another example, as describedfurther herein with regard to FIGS. 7 and 8, a first image frame sourcemay acquire image frames with different exposure settings and a firstframe rate, while a second image frame source may acquire image frameswith consistent exposure settings and a second frame rate higher thanthe first frame rate. As yet another example, as described furtherherein with regard to FIGS. 9-12, a plurality of image frame sources mayacquire image frames at a same frame rate but with staggered initialacquisition times and different shutter angles.

Thus, in some examples, method 200 may acquire the image frames directlyfrom the plurality of image frame sources (e.g., in real-time). In otherexamples, the image frames may be acquired by the plurality of imageframe sources prior to the execution of method 200, and so method 200may acquire the image frames from non-volatile memory (e.g., not inreal-time).

At 210, method 200 processes the image frames based on the differentacquisition settings to obtain final image frame(s). For example, asdescribed further herein with regard to FIGS. 3-6, motion estimation maybe determined from high-frame-rate but low-resolution image frames, andthen used to interpolate the low-frame-rate but high-resolution imageframes into a high-frame-rate, high-resolution video. As anotherexample, as described further herein with regard to FIGS. 7 and 8,motion estimation may be determined from the high-frame rate imageframes with consistent exposure settings, which may then be used toadjust the image frames with different exposures for merging orinterpolating into a single high-dynamic-range image frame. As yetanother example, as described further herein with regard to FIGS. 9-12,motion estimation may be determined from the image frames of all imageframe sources, which may be used to improve or adjust temporal coherenceand motion blur as desired.

After processing the image frames at 210, method 200 continues to 215.At 215, method 200 outputs the final image frame(s). The final imageframe(s) may be output as a video, in examples wherein a plurality ofimage frames are obtained, or as a single image frame. The final imageframe(s) are output to at least one of a display such as display 130 anda memory 124.

At 220, method 200 determines whether the acquisition is complete. Ifthe acquisition is not complete (“NO”), method 200 returns to 205 tocontinue acquiring image frames with the plurality of image framesources. Once the acquisition is complete (“YES”), method 200 returns.Thus, the processing techniques described herein may be applied to imageframes in real-time, so that image frames from a plurality of imageframe sources may be processed to generate high-quality video,high-frame-rate video, high-dynamic-range video, and combinationsthereof (e.g., high-quality, high-frame-rate video, high-quality HDRvideo, and so on). It should be appreciated that while the methodsdescribed herein may be performed in real-time, the methods are alsoapplicable for post-processing of image frames during post-production(i.e., after image frames are acquired and stored in non-volatilememory).

FIG. 3 shows a high-level flow chart illustrating an example method 300for image processing image frames from two image frame sources withdifferent acquisition parameters according to an embodiment. Method 300is described with regard to the systems and components of FIG. 1, thoughit should be appreciated that the method 300 may be implemented withother systems and components without departing from the scope of thepresent disclosure. Method 300 may be implemented as instructions inmemory 124, for example, executable by the image processor 120 toperform the actions described herein.

Method 300 begins at 302. At 302, method 300 receives a selection ofpreset motion appearance settings. For example, such preset motionappearance settings may be selected by a user according to the subjectbeing imaged and/or the environment in which the subject is being imaged(e.g., sports or high-speed action, cinematic, low-light, landscape,portrait, and so on).

Continuing at 305, method 300 acquires image frames via a first imageframe source with a low resolution and a high frame rate. For example,method 300 may control the first image frame source, such as the imageframe source 112, to capture or acquire image frames with low imageresolution and the high frame rate.

Meanwhile, at 310, method 300 acquires image frames via a second imageframe source with a high resolution and a low frame rate. For example,method 300 may control the second image frame source, such as the imageframe source 114, to capture or acquire image frames with high imageresolution but a low frame rate. As an example, the image frames may becaptured at a slower shutter speed to improve the signal-to-noise ratio,especially for low-light captures.

The initial starting time for acquiring the image frames with the firstimage frame source and the second image frame source may be controlledby the preset motion appearance settings selected at 302. For example,the starting times may be determined to define the base motion sampleperiod, temporal coherence, shutter angle, and motion linearity. Thestarting time(s) may thus be adjusted according to different temporalcoherence settings, where the higher the temporal coherence the largerthe starting time interval between the initial captures.

At 315, method 300 aligns the image frames from the first and secondimage frame sources. For example, method 300 may time align orsynchronize the timing of the image frames such that at least one imageframe from the first image source is time-aligned with a correspondingimage frame from the second image source. Further, as the image framesources are physically distinct and thus physically separate, method 300accounts for parallax when aligning the image frames based on knownrelative positioning of the first and second image frame sources. Insome examples, method 300 uses global statistics to determinemisalignment between image frames from the different image framesources, and corrects the misalignment between the image framesaccordingly.

Continuing at 320, method 300 down-samples the high-resolution imageframes to match the resolution of the low-resolution image frames. Then,at 325, method 300 searches the down-sampled image frames for objectmatches. Method 300 may use the object matches found in the down-sampledimage frames in order to improve the resolution.

At 330, method 300 performs motion model analysis for the low-resolutionimage frames to obtain motion vectors. Various types of motion modelsmay be used to obtain the motion vectors. For example, method 300determines motion vectors that describe the transformation from a firstimage frame of the low-resolution image frames to a second, subsequentimage frame of the low-resolution image frames. Method 300 may determinethe motion vectors for the regions of the image frames corresponding tothe object matches found in the down-sampled image frames at 325 as wellas block-based matches. In other examples, the motion estimation may beperformed globally for the image frames.

At 335, method 300 determines one or more deconvolution kernels toadjust the motion blur based on the motion vectors. For example, thedeconvolution kernel may be determined to reduce or remove motion bluror to introduce blur if additional blur is desired. Deconvolutionkernels may be determined or adjusted between image frames based on themotion vectors. At 340, method 300 applies the deconvolution kernel(s)to the image frames to adjust the motion blur.

Thus, method 300 uses the high-frame-rate image frames to create amotion model of objects captured at a lower frame rate or a longerexposure, and then estimates how the objects are blurred due to themotion and the camera motion. Method 300 then generates a kernelcorresponding to the blurring and sharpens the image frames with thekernel. Further, such blur kernels may be adjusted to change theeffective shutter speed of the low-frame-rate high-resolution imageframes.

At 345, method 300 generates final image frames with the selected presetmotion appearance based on the high-resolution image frames, thelow-resolution image frames, and the motion vectors. For example, method300 may interpolate the high-resolution image frames to match the framerate of the low-resolution image frames, with motion correction providedvia the motion vectors. As another example, the phases of the finalimage frames may be selected based on the selected preset motionappearance, and the high-resolution image frames and low-resolutionimage frames may be interpolated to generate the final image frames atthe phases, with adjustments to the motion vectors for phases of thefinal image frames that are not aligned with the phases of thelow-resolution image frames. An example of how the sampling or phases ofthe final image frames may be adjusted for different motion appearancesettings is discussed further herein with regard to FIG. 4.

At 350, method 300 outputs the final image frames. For example, method300 may output the final image frames to the display 130. The finalimage frames are displayed as a video with the desired motion appearancevia the display 130. Method 300 then returns.

In some examples, the first image frame source and the second imageframe source may comprise different types of cameras. In such examples,the first image frame source may acquire the image frames according tophases as preset by the camera type, while the second image frame sourcemay acquire higher-quality images based on a preset blur angle. Thefirst camera may be used to create motion vectors, and these motionvectors may be used to convert the output of the second camera to thepreset phases of the first camera. Depth information may be used tofurther refine the motion vectors when the first and second camerasampling time is the same. Thus the shutter angle, which defines theappearance of a video, may be set by the second, high-resolution camera,while the judder level may be adjusted by performing motioninterpolation with the motion vectors between image frames captured bythe first, lower-resolution camera.

As an illustrative example, FIG. 4 shows a high-level block diagramillustrating an example method 400 for leveraging motion estimation fromhigh-frame-rate image frames to increase frame rate for high-resolutionimage frames according to an embodiment. For example, method 400visually depicts an example of method 300 wherein the desired outputcomprises high-resolution, high-frame rate video.

A first image frame source, such as the image frame source 112, acquiresa first plurality of image frames 410 with a high frame rate and lowresolution. The first plurality of image frames 410 includes a firstimage frame 411, a second image frame 412, a third image frame 413, afourth image frame 414, a fifth image frame 415, and a sixth image frame416, as depicted.

A second image frame source, such as the image frame source 114,acquires a second plurality of image frames 420 with a low frame rateand a high resolution. The second plurality of image frames 422 includesa first image frame 421 and a second image frame 422.

While the image frames 410 are depicted as equally spaced in FIG. 4, itshould be appreciated that in general the image frames 410 may have anytemporal spacing between captures. Similarly, the image frames 420 mayalso be captured with equal or unequal sampling times. Further still,the first image frame 411 and the first image frame 421, may be capturedat a same time in some examples, or may be captured at slightlydifferent times in other examples. Similarly, the sixth image frame 416and the second image frame 422 may be captured at a same time or atslightly different times.

Method 400 determines motion vectors between the image frames of thefirst plurality of image frames 410, including a motion vector 432between the first image frame 411 and the second image frame 412, amotion vector 434 between the second image frame 412 and the third imageframe 413, and a motion vector 436 between the first image frame 411 andthe third image frame 413. It should be appreciated that additionalmotion vectors are determined, including additional motion vectorscharacterizing motion of different objects between frames, and that themotion vectors depicted are illustrative and non-limiting.

Further, method 400 aligns 430 the camera outputs such that the firstimage frame 411 is aligned with the first image frame 421, and furthersuch that the sixth image frame 416 is aligned with the second imageframe 422. Method 400 then interpolates 438 the high-resolution imageframes 420, aided by the motion vectors including motion vector 432determined from the low-resolution image frames 410, to generate aplurality of final image frames 440 including a first final image frame441, a second final image frame 442, a third final image frame 443, afourth final image frame 444, a fifth final image frame 445, and a sixthfinal image frame 446. In some examples, the first final image frame 441comprises the image frame 421, and the sixth final image frame 446comprises the image frame 422.

The sampling of the final image frames 440, or the time intervalsbetween each frame, may be determined or selected based on a presetmotion appearance or a desired motion appearance. For example, a presetmotion appearance may comprise a “Sports” motion appearance that may beselected for video content including rapidly-moving objects. For such apreset motion appearance, the sampling of the final image frames 440 maybe equal, such that each image frame of the final image frames 440 isequally spaced. In this way, the final motion appearance of the finalimage frames 440 displayed as a video depicts smooth motion.

As another example, the preset motion appearance for a “Cinematic”motion appearance may be selected such that the final output videoexhibits a “cinematic feel” that matches the look and feel of a movie orfilm played in a cinema. To obtain the cinematic feel with a reductionin judder, the sampling of the final image frames 440 may be unequal. Insuch an example, the output image frames 440 may include at least oneimage frame that is not time-aligned with any of the image frames fromthe first plurality of image frames 410 or the second plurality of imageframes 420. As depicted, the third final image frame 443 is positionedbetween the second image frame 412 and the third image frame 413 of thefirst plurality of image frames 410. To properly interpolate the thirdfinal image frame 443, the motion vector 432 between the first imageframe 411 and the second image frame 412 may be combined with a fractionof the motion vector 434 between the second image frame 412 and thethird image frame 413. For example, expressing the motion vector 423between the first image frame 411 and the second image frame 412 asMV12, and the motion vector 434 between the second image frame 412 andthe third image frame 413 as MV23, a candidate for the motion vector 436between the first image frame 411 and the third image frame 413, whichmay be expressed as MV13, may include a vector addition of the motionvectors 432 and 434:

MV13=MV12+MV23.

To interpolate the third final output frame 443 at a time interval orsampling between the second image frame 412 and the third image frame413, a motion vector may comprise an adaptation of the motion vector436:

MV13′=MV12+k*MV23,

where k is a parameter between 0 and 1 for scaling the motion vector 434or MV23, which may be determined according to the preset motionappearance to reduce judder in the final output image frames 440.

Further, if the output is at a lower frame rate than the high frame rateof the image frames 410, then the motion vectors for the image frames410 would be converted to motion vectors at the lower frame rate. As themotion estimation involves tracking objects/blocks, motion vectors for anumber of frames would thus be combined to track the object over alonger time period.

FIG. 5 shows a graph 500 illustrating example acquisition timings ofimage frames with different image frame sources according to anembodiment. The graph 500 depicts acquisition timings for a first set ofimage frames 510, which may correspond to the first set of image frames410 described hereinabove, as well as acquisition timings for a secondset of image frames 520, which may correspond to the second set of imageframes 420. That is, the first set of image frames 510 compriselow-resolution, high-frame-rate image frames while the second set ofimage frames 520 comprise high-resolution, low-frame-rate image frames.As depicted, the acquisitions may be synchronized such that each imageframe of the second set of image frames 520 coincides with anacquisition of a corresponding image frame in the first set of imageframes 510. However, in some examples, the acquisitions may beasynchronous or staggered such that high-resolution image frames andlow-resolution image frames are not acquired at a same time. In suchexamples, the asynchronicity may be intentionally selected based on thepreset motion appearance settings. Otherwise, the first set of imageframes 510 or the second set of image frames 520 may be adjusted suchthat the image frames are time aligned.

Further, the exposure settings of the different image frames may bedifferent. For example, as depicted, the exposure time for each imageframe of the first set of image frames 510 is shorter than the exposuretime for each image frame of the second set of image frames 520.

FIG. 6 shows a high-level block diagram illustrating an example dataflow 600 for image processing image frames from two image frame sourcesaccording to an embodiment. The data flow 600 depicts how low-resolution(LR) image frames 610 acquired at a high frame rate and high-resolution(HR) image frames 620 acquired at a low frame rate are processed togenerate high-resolution, high-frame-rate video. The data flow 600 istherefore an example implementation of the method 300 describedhereinabove for image processing image frames from two image framesources with different acquisition parameters.

As depicted, motion analysis 630 is performed on the low-resolutionimage frames 610 to determine object motion (OM) 632 between thelow-resolution image frames 610. The object motion 632 may berepresented or characterized by one or more motion vectors between theimage frames 610.

The object motion 632 is then use for blur analysis 660 to determine atleast one blur kernel 662. For example, the blur analysis 660 determinesone or more deconvolution kernels or blur kernels 662 to adjust themotion blur based on the motion vectors describing the objection motion632. For example, the deconvolution or blur kernel may be determined toreduce or remove motion blur or to introduce blur if additional blur isdesired. As depicted, the blur kernel 662 is applied to thehigh-resolution image frames 620 for blur control 665, thereby resultingin a first corrected high-resolution image frame (HR′) 667.

Meanwhile, the low-resolution image frames 610 are up-sampled 640 toobtain up-sampled low-resolution image frames (LR′) 642. In particular,the low-resolution image frames 610 may be up-sampled to match theresolution of the high-resolution image frames 620. A match search 650is applied to the up-sampled low-resolution image frames 642 as well asthe high-resolution image frames 620 to identify matching blocks betweenthe high-resolution image frames. The match search 650 produces one ormore fusion parameters 652 for fine-tuning the alignment between thelow-resolution image frames 610 and the high-resolution image frames620.

Further, the low-resolution image frames 610, the object motion 632measured therefrom, and the first corrected high-resolution image frames667 are input to a cover/uncover/recover module 670 for detectingcovered, uncovered, or recovered regions. Cover/uncover/recoverdetection involves determining the regions, if any, in which a movingobject either covers something that was previously uncovered, uncoversbackground that was previously covered, or recovers something that waspreviously covered and then uncovered. Such regions are referred to asoccluded regions. The cover/uncover/recover module 670 thus compares theimage frames 610 and 667, for example on a block-by-block basis, toidentify occluded regions and correct motion vectors linked to theoccluded regions. The cover/uncover/recover module 670 outputs secondcorrected high-resolution image frames (HR″) 672.

The low-resolution image frames 610, the high-resolution image frames620, the second corrected high-resolution image frames 672, and thefusion parameter(s) 652 are then input to a resolution enhance/noisereduction module 680 which performs resolution enhancement, noisereduction, and interpolation based on the various inputs to produce aplurality of high-resolution, high-frame rate (HR HFR) image frames 682.The high-resolution, high-frame rate image frames 682 are output to oneor more of a memory (e.g., memory 124), for storage and subsequentretrieval, and a display (e.g., display 130), for displaying the videocomprising the plurality of high-resolution, high-frame-rate imageframes 682.

As another example of leveraging information from multiple image framesources to improve image processing, FIG. 7 shows a high-level flowchart illustrating an example method 700 for generating ahigh-dynamic-range (HDR) image based on image frames acquired withmultiple image frame sources according to an embodiment. Method 700 isdescribed with regard to the systems and components of FIG. 1, though itshould be appreciated that the method 700 may be implemented with othersystems and components without departing from the scope of the presentdisclosure. Method 700 may be implemented as instructions in memory 124,for example, executable by the image processor 120 to perform theactions described herein.

Method 700 begins at 705. At 705, method 700 acquires image frames witha first image frame source with different exposures. For example, thefirst image frame source, such as the image frame source 112, acquires aplurality of image frames with different exposures. In some examples,the first image frame source may alternate between a first set ofexposure settings (e.g., long exposure settings) and a second set ofexposure settings (e.g., short exposure settings) to acquire alternatinglong-exposure image frames and short-exposure image frames. In otherexamples, the first image frame source may alternate between more thantwo sets of exposure settings. For example, a third set of exposuresettings (e.g., medium exposure settings) may be included such that thefirst image frame source acquires alternating long, medium, andshort-exposure image frames over time.

Meanwhile, at 710, method 700 acquires image frames with a second imageframe source with a high frame rate, low resolution, and a sameexposure. Method 700 may acquire the image frames with the second imageframe source, such as the image frame source 114, while also acquiringthe image frames from the first image frame source.

At 715, method 700 aligns the image frames of the first image framesource and the second image frame source. Further, at 720, method 700performs motion model analysis of the image frames from the second imageframe source to determine motion vectors. Method 700 may further accountfor the relative positioning of the first and second image framesources, for example, to correct parallax errors.

At 725, method 700 interpolates the image frames from the first imageframe source based on the motion vectors to generate a combined HDRimage. By using the motion vectors acquired from the image frames of thesecond image frame source, the alignment and interpolation of the imageframes from the first image frame source is improved. If the captureblur of the image frames is not equalized (e.g., via a deconvolution orblur kernel) and aligned, the shorter exposure looks sharper and offsetfrom the longer exposure. Therefore, optionally, a deconvolution kernelor blur kernel may be determined from the motion vectors as describedhereinabove, and the kernel is applied to adjust the blur of the imageframes from the first image frame source so that when combined allobjects within the image frames have the same look. To interpolate theimage frames, the image frames may be selectively merged together. Forexample, pixel values of the long exposure image frames may be reducedso that they represent the same scene luminance values that are capturedby the short exposure image frames. When the long exposure image frameis saturated and no longer able to capture the details for high lightareas, the pixels from shorter exposure image frames are used to fill inthe details. At 730, method 700 outputs the combined image. For example,method 700 may output the combined image to the display 130. Method 700then returns.

FIG. 8 shows a high-level block diagram illustrating a method 800 forleveraging image frames with a same exposure to improve interpolation ofimage frames with different exposures to generate a high-dynamic-rangeimage according to an embodiment. For example, method 800 visuallydepicts an example of method 700 wherein several different exposuresettings are used.

As depicted, a first set of image frames 810 including a first imageframe 812, a second image frame 814, and a third image frame 816 isacquired with a first image frame source, such as the image frame source112. The first image frame 812 is acquired with a long exposure, thesecond image frame 814 is acquired with a medium exposure, and the thirdimage frame 816 is acquired with a short exposure.

Meanwhile, a second set of image frames 820 including a first imageframe 822, a second image frame 823, a third image frame 824, a fourthimage frame 825, and a fifth image frame 826 are acquired with a second,different image frame source, such as the image frame source 114. Theimage frames 820 are acquired with a same exposure and a low resolution.Further, the image frames 820 are acquired with a higher frame rate incomparison to the frame rate of the image frames 810 from the firstimage frame source, as depicted. In this way, motion estimation may beperformed with the second set of image frames 820 which may be leveragedfor blur correction of the first set of image frames 810, for example asdescribed hereinabove with regard to FIG. 3.

As depicted, the outputs between the different image frame sources arealigned 830 at least between the medium-exposure image frame 814 and thethird image frame 824, based on the assumption that the exposure of thethird image frame 824 is the same as the exposure of the medium-exposureimage frame 814. In general, frames with a same exposure from thedifferent image frame sources are aligned. The second set of imageframes 820 is used for motion estimation over time, which may bedifficult to perform for the first set of image frames 810 given thedifferent exposure rates. The first set of image frames 810 areinterpolated 835 based on the motion estimated from the second set ofimage frames 820 to generate a combined HDR image 840.

It should be appreciated that while FIGS. 7 and 8 are described withregard to obtaining a single combined HDR image, the methods may beexecuted repeatedly to acquire HDR video. For example, method 700 may berepeated to generate a plurality of combined image frames, which may bedisplayed with the resulting frame rate as HDR video.

In some examples, more than two image frame sources may be used toacquire image frames, and the data from each stream of image frames fromthe multiple image frame sources may be leveraged to produce an improvedfinal output. As an illustrative example, FIG. 9 shows a high-level flowchart illustrating an example method 900 for generating high-frame-ratevideo from output of a plurality of image frame sources acquiring at lowframe rates according to an embodiment. Method 900 is described withregard to the systems and components of FIG. 1, though it should beappreciated that the method 900 may be implemented with other systemsand components without departing from the scope of the presentdisclosure. Method 900 may be implemented as instructions in memory 124,for example, executable by the image processor 120 to perform theactions described herein.

Method 900 begins at 905. At 905, method 900 acquires image frames fromtwo or more image frame sources with a same frame rate and staggeredacquisition times. As an example, the image frame sources may acquireimage frames at a frame rate of 24 frames per second. For staggeredacquisition times, the initial starting acquisition time for each imageframe source is delayed with respect to each other such that none of theimage frame sources simultaneously acquire image frames. The distanceintervals between the starting times are controllable or adjustable toadjust the smoothness of the final video. For example, the intervals maybe equal so that the acquisitions are evenly distributed over time.Alternatively, one or more of the intervals may be increased to increasethe smoothness of the final video, or decreased to decrease thesmoothness.

At 910, method 900 arranges the image frames from all of the image framesources into a single set of image frames according to the order ofacquisition. Continuing at 910, method 900 analyzes the set of imageframes from all of the image frame sources to generate motion vectors.As the different image frame sources or cameras may exhibit tilt error,wrong start timing, different lighting conditions, and so on, motionestimation and motion compensation may be applied to the full set ofimage frames in order to minimize artifacts or substantialdiscontinuities (e.g., flashing) between image frames. Then, at 915,method 900 applies motion correction to the set of image frames based onthe motion vectors to generate final image frames.

At 920, method 900 outputs the final image frames. For example, method900 may output the final image frames to the display 130. The finalimage frames are displayed as a video with a higher frame rate than theframe rate of the individual image frame sources. For example, for amulti-camera system including five image frame sources, if the framerate of the acquisition for each image frame source is 24 frames persecond, the frame rate of the final image frames is 120 frames persecond. Method 900 then returns.

The initial start time for acquiring image frames for each image framesource may be selected to determine the appearance of the final imageframes or the final video. As an illustrative example, FIG. 10 shows atiming diagram 1000 illustrating acquisition times for a plurality ofimage frame sources to generate high-frame-rate video according to anembodiment. In particular, the timing diagram 1000 illustrates theacquisition times for five cameras or five image frame sources,including a zeroth camera (Cam 0), a first camera (Cam 1), a secondcamera (Cam 2), a third camera (Cam 3), and a fourth camera (Cam 4).

As depicted by the acquisition timing 1002 for the zeroth camera, thezeroth camera acquires a first image frame 1004 at time t0, a secondimage frame 1005 at time t5, and a third image frame 1006 at time t10.The interval 1008 between the first image frame 1004 and the secondimage frame 1005 establishes the base motion sample frequency.

As depicted by the acquisition timing 1012 for the first camera, thefirst camera acquires a first image frame 1014 at time t1, and a secondimage frame 1015 at time t6. The interval 1018 between the acquisitionof the first image frame 1004 by the zeroth camera and the acquisitionof the first image frame 1014 by the first camera establishes the motionlinearity.

As depicted by the acquisition timing 1022 for the second camera, thesecond camera acquires a first image frame 924 at time t2, and a secondimage frame 1025 at time t7. The interval 1028 between the acquisitionof the first image frame 1014 by the first camera and the acquisition ofthe first image frame 1024 by the second camera, along with the interval1018, further determines the motion linearity.

As depicted by the acquisition timing 1032 for the third camera, thethird camera acquires a first image frame 1034 at time t3, and a secondimage frame 1035 at time t8. The interval 1038 between the acquisitionof the first image frame 1024 by the second camera and the acquisitionof the first image frame 1034 by the third camera defines the level ofcoherence or the level of smoothness of the final video.

Further, as depicted by the acquisition timing 1042 for the fourthcamera, the fourth camera acquires a first image frame 1044 at time t4,and a second image frame 1045 at time t9. The timing for acquiring thefirst image frame 1044 may be based on the interval 1028, in someexamples, such that the interval between time t3 and time t4 equals theinterval 1028 between time t1 and time t2. However, the acquisitiontiming 1042 for the fourth camera may be adjusted to control thetemporal coherence, which is set by the interval 1048 between the timet4 and the time t5.

Thus, the interval 1008 determines the base motion sample frequency, theintervals 1014 and 1018 define the motion linearity, and the interval1038 defines the level of coherence for the final video comprising afusion of the image frames from all of the image frame sources. Theintervals may be adjusted according to a desired appearance of the finalvideo.

FIG. 11 shows a high-level block diagram illustrating an example dataflow 1100 for managing multiple image frame streams with adjustments fortemporal coherence and motion blur according to an embodiment. Inparticular, the data flow 1100 depicts the management of multiple imageframe streams from a multi-camera system or a plurality of image framesources 1105 including a zeroth camera (C0) or zeroth image source 1110,a first camera (C1) or first image source 1111, a second camera (C2) orsecond image source 1112, a third camera (C3) or third image source1113, and a fourth camera (C4) or fourth image source 1114. Each cameraor image frame source of the plurality of image frame sources 1105 maycapture image frames at a same frame rate, for example as describedhereinabove, with different acquisition settings.

For example, the image frame sources 1105 may be controlled according totiming control inputs 1106 which include a first time interval dt1 thatcontrols an initial acquisition time of the first image frame source1111 relative to an initial acquisition of the zeroth image frame source1110, a second time interval dt2 that controls an initial acquisitiontime of the second image frame source 1112 relative to the initialacquisition time of the first image frame source 1111, a third timeinterval dt3 that controls an initial acquisition time of the thirdimage frame source 1113 relative to the initial acquisition time of thesecond image frame source 1112, and a fourth time interval dt4 thatcontrols an initial acquisition time of the fourth image frame source1114 relative to the initial acquisition time of the third image framesource 1113.

The image frame sources 1105 may be further controlled according toshutter angle control inputs 1108 for controlling the shutter angle ofeach image frame source. For example, the shutter angle control inputs1108 may include a zeroth shutter angle for the zeroth image framesource 1110, a first shutter angle for the first image frame source1111, a second shutter angle for the second image frame source 1112, athird shutter angle for the third image frame source 1113, and a fourthshutter angle for the fourth image frame source 1114. The timing controlinputs 1106 and the shutter angle control inputs 1108 may be selectedaccording to a desired motion appearance, as described hereinabove.

The plurality of image frame sources 1105 outputs a correspondingplurality of low-frame-rate image frame streams 1120 to a multi-streammanagement module 1130. The multi-stream management module 1130interleaves the plurality of image frame streams 1120 to generate asingle high-frame-rate image frame stream 1132. For example, if the lowframe rate of each image frame source of the plurality of image framesources 1105 is 24 fps, then the multi-stream management 1130 combinesall of the image frames into a single stream of image frames with aframe rate of 120 fps, since five image frame sources multiplied by 24fps per source results in 120 fps. The frame rate of the single imageframe stream 1132 thus depends on the number of image frame sources inthe plurality of image frame sources 1105 as well as the original framerate for each image frame source. While five image frame sources aredepicted as acquiring image frames at 24 fps, it should be appreciatedthat a different number of image frame sources may be used to acquireimage frames at a different frame rate than 24 fps. The particularsettings depend on the configuration of the image frame sources (e.g.,the number of image frame sources) as well as the desired final framerate of the single image frame stream 1132.

The multi-stream management module 1130 also outputs the image framestreams 1120 and/or the single image frame stream 1132 to a motionanalysis module 1140 for analyzing motion between the image frames. Themotion analysis module 1140 may analyze motion between image frames inindividual stream of the plurality of image frame streams 1120, forexample. Additionally or alternatively, the motion analysis module 1140may analyze motion between image frames in the single image frame stream1132. The motion analysis module 1140 generates a motion model 1142comprising, for example, motion vectors depicting motion of objects on ablock-by-block and/or a global level between image frames.

The motion model 1140 and the single image frame stream 1132 are inputto a temporal coherence adjustment module 1150 which may adjust thetemporal coherence of the single image frame stream 1132, for example,based on the motion model 1142. For example, the temporal coherenceadjustment module 1150 may adjust the phases of one or more image framesin the single image frame stream 1132 by adjusting or scaling the motionvectors of the motion model 1142, as described hereinabove. In someexamples, the temporal coherence adjustment module 1150 may not applytemporal coherence adjustments if the temporal coherence of the singleimage frame stream 1132 already corresponds to a desired temporalcoherence.

The single image frame stream 1132, after optional temporal coherenceadjustments by the temporal coherence adjustment module 1150, is inputalong with the motion model 1142 to a motion blur adjustment module1160. The motion blur adjustment module 1160 determines one or moredeconvolution or blur kernels based on the motion model 1142 and appliesthe kernel(s) to the image frames of the single image frame stream 1132.The motion blur adjustment module 1160 thus outputs a high-frame-ratesingle image frame stream 1162. Thus, a plurality of low-frame-rateimage frame streams may be combined into a single high-frame-rate imageframe stream, with temporal coherence adjustments and blur adjustmentsto achieve a high-frame-rate video with a desired motion appearance anda consistent look.

In some examples, multiple image frame streams may be integrated into asingle image frame stream without additional adjustments to temporalcoherence and blur. As an illustrative example, FIG. 12 shows ahigh-level block diagram illustrating an example data flow 1200 formanaging multiple image frame streams according to an embodiment. Inparticular, similar to the data flow 1100, the data flow 1200 depictsthe management of multiple image frame streams from a multi-camerasystem or a plurality of image frame sources 1205 including a zerothcamera (C0) or zeroth image source 1210, a first camera (C1) or firstimage source 1211, a second camera (C2) or second image source 1212, athird camera (C3) or third image source 1213, and a fourth camera (C4)or fourth image source 1214. Each camera or image frame source of theplurality of image frame sources 1205 may capture image frames at a sameframe rate, for example as described hereinabove, with differentacquisition settings.

For example, the image frame sources 1205 may be controlled according totiming control inputs 1206 which include a first time interval dt1 thatcontrols an initial acquisition time of the first image frame source1211 relative to an initial acquisition of the zeroth image frame source1210, a second time interval dt2 that controls an initial acquisitiontime of the second image frame source 1212 relative to the initialacquisition time of the first image frame source 1211, a third timeinterval dt3 that controls an initial acquisition time of the thirdimage frame source 1213 relative to the initial acquisition time of thesecond image frame source 1212, and a fourth time interval dt4 thatcontrols an initial acquisition time of the fourth image frame source1214 relative to the initial acquisition time of the third image framesource 1213.

The image frame sources 1205 may be further controlled according toshutter angle control inputs 1208 for controlling the shutter angle ofeach image frame source. For example, the shutter angle control inputs1208 may include a zeroth shutter angle for the zeroth image framesource 1210, a first shutter angle for the first image frame source1211, a second shutter angle for the second image frame source 1212, athird shutter angle for the third image frame source 1213, and a fourthshutter angle for the fourth image frame source 1214. The timing controlinputs 1206 and the shutter angle control inputs 1208 may be selectedaccording to a desired motion appearance, as described hereinabove.

The plurality of image frame sources 1205 outputs a correspondingplurality of low-frame-rate image frame streams 1220 to a multi-streammanagement module 1230. The multi-stream management module 1230interleaves the plurality of image frame streams 1220 to generate asingle high-frame-rate image frame stream 1232. The look and feel of thevideo corresponding to the single high-frame-rate image frame stream1232 is thus established by the control inputs 1206 and 1208.

FIG. 13 shows a diagram 1300 illustrating motion vectors between imageframes acquired by a plurality of image frame sources to generatehigh-frame-rate video according to an embodiment. In particular, thediagram 1300 illustrates the acquisition times described hereinabovewith regard to FIG. 10 for five cameras or five image frame sources,including a zeroth camera (Cam 0), a first camera (Cam 1), a secondcamera (Cam 2), a third camera (Cam 3), and a fourth camera (Cam 4). Asdepicted, the diagram 300 includes the acquisition timings 1002, 1012,1022, 1032, and 1042 for the zeroth camera, the first camera, the secondcamera, the third camera, and the fourth camera, respectively.

For the image frames depicted according to the different acquisitiontimings, a plurality of motion vectors may be calculated. The pluralityof motion vectors may include motion vectors between image frames foreach image frame source or camera. For example, the plurality of motionvectors may include a motion vector 1307 between the image frame 1004and the image frame 1005 for the zeroth camera, a motion vector 1317between the image frame 1014 and the image frame 1015 for the firstcamera, a motion vector 1327 between the image frame 1024 and the imageframe 1025 for the second camera, a motion vector 1337 between the imageframe 1034 and the image frame 1035 for the third camera, and a motionvector 1347 between the image frame 1044 and the image frame 1045 forthe fourth camera.

As discussed hereinabove, the timing of the acquisition for each imageframe source may be adjusted to control the overall look and feel of thefinal fused video. In some examples, the intervals for each image framesource may be determined relative to the acquisition of the zerothcamera. For example, the time t1 for acquiring the first image frame1014 via the first camera may be equal to t0+dt1, where dt1 is theinterval between t0 and t1. Similarly, the time t2 may be expressed ast0+dt2, the time t3 may be expressed as t0+dt3, the time t4 may beexpressed as t0+dt4, the time t6 may be expressed as t5+dt1, the time t7may be expressed as t5+dt2, and so on.

These motion vectors between image frames for each image frame sourcemay be used to generate or interpolate intermediate image frames betweenthe acquired image frames. For example, the motion vector 1307 may beused to calculate motion vectors for four phases between the first imageframe 1004 and the second image frame 1005. These calculated motionvectors may then be used to generate an image frame for each of the fourphases based on the first image frame 1004 and the second image frame1005. The four phases may be determined based on the intervals betweenimage acquisitions for all of the cameras or image frame sources. Forexample, the phases between time t0 and time t5 for the intermediateimage frames between image frames 1004 and 1005 may comprisedt1/(t5−t0), dt2/(t5−t0), dt3/(t5−t0), and dt4/(t5−t0).

Similarly, the motion vector 1317 for the first camera may be used tocalculate motion vectors for four phases between the first image frame1014 and the second image frame 1015 for the first camera. Thesecalculated motion vectors may then be used to generate an image framefor each of the four phases based on the first image frame 1014 and thesecond image frame 1015. The four phases may be determined based on theintervals. For example, the phases between time t0 and time t5 for theintermediate image frames between image frames 1014 and 1015 maycomprise (dt2−dt1)/(t5−t0), (dt3−dt2)/(t5−t0), (dt4−dt3)/(t5−t0), and(t5−t0−dt1)/(t5−t0).

The motion vector 1327 for the second camera may be used to calculatemotion vectors for four phases between the first image frame 1024 andthe second image frame 1025 for the second camera. These calculatedmotion vectors may then be used to generate an image frame for each ofthe four phases based on the first image frame 1024 and the second imageframe 1025. For example, the phases between time t0 and time t5 for theintermediate image frames between image frames 1024 and 1025 maycomprise (dt3−dt2)/(t5−t0), (dt4−dt3)/(t5−t0), (t5−t0−dt2)/(t5−t0), and(t5−t0−(dt2−dt1))/(t5−t0).

The motion vector 1337 for the third camera may be used to calculatemotion vectors for four phases between the first image frame 1034 andthe second image frame 1035 for the third camera. These calculatedmotion vectors may then be used to generate an image frame for each ofthe four phases based on the first image frame 1034 and the second imageframe 1035. For example, the phases between time t0 and time t5 for theintermediate image frames between image frames 1034 and 1035 maycomprise (dt4−dt3)/(t5−t0), (t5−t0−dt3)/(t5−t0),(t5−t0−(dt3−dt1))/(t5−t0), and (t5−t0−(dt3−dt2))/(t5−t0).

Further, the motion vector 1347 for the fourth camera may be used tocalculate motion vectors for four phases between the first image frame1044 and the second image frame 1045 for the fourth camera. Thesecalculated motion vectors may then be used to generate an image framefor each of the four phases based on the first image frame 1044 and thesecond image frame 1045. For example, the phases between time t0 andtime t5 for the intermediate image frames between image frames 1044 and1045 may comprise (t5−t0−dt4)/(t5−t0), (t5−t0−(dt4−dt1))/(t5−t0),(t5−t0−(dt4−dt2))/(t5−t0), and (t5−t0−(dt4−dt3)/(t5−t0).

The plurality of motion vectors may further include motion vectorsbetween image frames from different image frame sources. For example, asdepicted, the plurality of motion vectors may include a motion vector1309 between the image frame 1004 from the zeroth camera and the imageframe 1014 from the first camera, a motion vector 1329 between the imageframe 1014 from the first camera and the image frame 1024 from thesecond camera, a motion vector 1339 between the image frame 1024 fromthe second camera and the image frame 1034 from the third camera, and amotion vector 1349 between the image frame 1034 from the third cameraand the image frame 1044 from the fourth camera. These motion vectorsbetween image frames from different image frame sources may be used toaccount for potential image shifts between the cameras. For example, themotion vector 1309 calculated between image frame 1004 from the zerothcamera and image frame 1014 from the first camera is a motion vectorfield that may be used to account for a difference in positions of thezeroth camera and the first camera as well as image shifts between theimage frames 1004 and 1014 in general. Similarly, the motion vector 1329calculated between image frame 1014 from the first camera and imageframe 1024 from the second camera is a motion vector field that may beused to account for potential image shifts between the first camera andthe second camera, the motion vector 1339 calculated between image frame1024 from the second camera and image frame 1034 from the third camerais a motion vector field that may be used to account for potential imageshifts between the second camera and the third camera, and the motionvector 1349 calculated between image frame 1034 from the third cameraand image frame 1044 from the fourth camera is a motion vector fieldthat may be used to account for potential image shifts between the thirdcamera and the fourth camera.

At each capture time, image fusion is applied based on these motionvector fields between the cameras. Further, these motion vector fieldsmay be approximated by using the global motion vector of the field. Thefinal video comprises a fusion of the image frames from all five imageframe sources.

Thus, various embodiments of image processing for multiple image framesources or cameras are provided. In one embodiment, a method comprisesacquiring image frames with a plurality of image frame sourcesconfigured with different acquisition settings, processing the imageframes based on the different acquisition settings to generate at leastone final image frame, and outputting the at least one final imageframe. The at least one final image frame is output to one or more of adisplay (e.g., for displaying the at least one final image frame) and amemory (e.g., to store the at least one final image frame for subsequentpost-processing and/or display).

In a first example of the method, acquiring the image frames with theplurality of image frame sources configured with different acquisitionsettings comprises acquiring a first set of image frames with a firstimage frame source and a second set of image frames with a second imageframe source, wherein the first set of image frames are acquired with alow resolution and a high frame rate, and the second set of image framesare acquired with a high resolution and a low frame rate. In a secondexample of the method optionally including the first example, the methodfurther comprises aligning at least one image frame from the first setof image frames with an image frame from the second set of image frames.In a third example of the method optionally including one or more of thefirst and second examples, processing the image frames based on thedifferent acquisition settings to generate the at least one final imageframe comprises estimating motion in the first set of image frames. In afourth example of the method optionally including one or more of thefirst through third examples, processing the image frames based on thedifferent acquisition settings to generate the at least one final imageframe further comprises sharpening the second set of image frames with adeconvolution kernel determined from the estimated motion in the firstset of image frames. In a fifth example of the method optionallyincluding one or more of the first through fourth examples, processingthe image frames based on the different acquisition settings to generatethe at least one final image frame further comprises interpolatinghigh-resolution image frames between the image frames of the second setof image frames based on the estimated motion to generate a set of finalimage frames. In a sixth example of the method optionally including oneor more of the first through fifth examples, acquiring the image frameswith the plurality of image frame sources comprises acquiring a firstset of image frames with a first image frame source, each image frame ofthe first set of image frames acquired with a different exposuresetting, and acquiring a second set of image frames with a second imageframe source, wherein the second set of image frames are acquired withconsistent exposure settings. In a seventh example of the methodoptionally including one or more of the first through sixth examples,processing the image frames based on the different acquisition settingsto generate at least one final image frame comprises estimating motionin the second set of image frames, and interpolating the first set ofimage frames based on the estimated motion in the second set of imageframes to generate the at least one final image frame. In an eighthexample of the method optionally including one or more of the firstthrough seventh examples, the method further comprises arranging theimage frames into a single set of image frames according to order ofacquisition of the image frames, each image frame source configured withan acquisition frame rate, wherein the different acquisition settingscomprises different initial start times for each image frame source, andwherein intervals between the initial start times for each image framesource are adjustable to control an appearance of a video formed fromthe single set of image frames. In a ninth example of the methodoptionally including one or more of the first through eighth examples,the method further comprises estimating motion between image frames inthe single set of image frames, and applying motion compensation to theimage frames based on the estimated motion to generate amotion-corrected set of image frames with a frame rate higher than theacquisition frame rate, wherein the at least one final image framecomprises the motion-corrected set of image frames.

In another embodiment, a method comprises acquiring a first set of imageframes with a first image frame source, acquiring a second set of imageframes with a second image frame source, the first set of image framesacquired in parallel with the second set of image frames, estimatingmotion in the second set of image frames, generating at least onecorrected image frame from the first set of image frames and theestimated motion, and outputting the at least one corrected image frameto a display.

In a first example of the method, the first image frame source acquiresthe first set of image frames with a high resolution and a low framerate, and wherein the second image frame source acquires the second setof image frames with a low resolution and a high frame rate. In a secondexample of the method optionally including the first example, the methodfurther comprises filtering the first set of image frames with adeconvolution kernel determined from the estimated motion to adjust blurin the first set of image frames. In a third example of the methodoptionally including one or more of the first and second examples, thefirst image frame source acquires the first set of image frames withalternating exposure settings, and the second image frame sourceacquires the second set of image frames with a single set of exposuresettings. In a fourth example of the method optionally including one ormore of the first through third examples, the method further comprisesaligning at least one image frame from the first set of image frameswith a corresponding image frame from the second set of image frames.

In yet another embodiment, a system comprises an image processorconfigured with instructions in non-transitory memory that when executedcause the image processor to: acquire, via a plurality of image framesources configured with different acquisition settings, a plurality ofimage frames; process the plurality of image frames based on thedifferent acquisition settings to generate at least one final imageframe; and output the at least one final image frame to a display.

In a first example of the system, to acquire the image frames with theplurality of image frame sources configured with different acquisitionsettings, the image processor is configured with instructions that whenexecuted cause the image processor to acquire a first set of imageframes with a first image frame source and a second set of image frameswith a second image frame source, wherein the first set of image framesare acquired with a low resolution and a high frame rate, and the secondset of image frames are acquired with a high resolution and a low framerate. In a second example of the system optionally including the firstexample, to process the image frames based on the different acquisitionsettings to generate the at least one final image frame, the imageprocessor is further configured with instructions that when executedcause the image processor to estimate motion in the first set of imageframes, and interpolate high-resolution image frames between the imageframes of the second set of image frames based on the estimated motionto generate a set of final image frames. In a third example of thesystem optionally including one or more of the first and secondexamples, to acquire the image frames with the plurality of image framesources, the image processor is configured with instructions that whenexecuted cause the image processor to acquire a first set of imageframes with a first image frame source, each image frame of the firstset of image frames acquired with a different exposure setting, andacquire a second set of image frames with a second image frame source,wherein the second set of image frames are acquired with consistentexposure settings. In a fourth example of the system optionallyincluding one or more of the first through third examples, to processthe plurality of image frames based on the different acquisitionsettings to generate at least one final image frame, the image processoris further configured with instructions that when executed cause theimage processor to estimate motion in the second set of image frames,and interpolate the first set of image frames based on the estimatedmotion in the second set of image frames to generate the at least onefinal image frame.

In one representation, a method comprises acquiring a plurality of imageframes from each image frame source of a plurality of image framesources, each plurality of image frames acquired at a first frame rate,combining the plurality of image frames from each image frame sourceinto a single plurality of image frames with a second frame rate higherthan the first frame rate, and outputting the single plurality of imageframes as a video.

In a first example of the method, the plurality of image frame sourcescomprises a first image frame source, a second image frame source, athird image frame source, a fourth image frame source, and a fifth imageframe source, wherein the plurality of image frame sources arepositioned adjacent to each other and oriented with overlapping fieldsof view. In a second example of the method optionally including thefirst example, the first frame rate comprises 24 frames per second andthe second frame rate comprises 120 frames per second. In a thirdexample of the method optionally including one or more of the first andsecond examples, the method further comprises staggering an initialacquisition time for each image frame source such that at most one imageframe source acquires an image frame at a time. In a fourth example ofthe method optionally including one or more of the first through thirdexamples, relative intervals between the initial acquisition times forthe plurality of image frame sources are adjustable to control motionlinearity of the single plurality of image frames. In a fifth example ofthe method optionally including one or more of the first through fourthexamples, the method further comprises calculating motion vectorsbetween image frames in each plurality of image frames and between imageframes from different image frame sources. In a sixth example of themethod optionally including one or more of the first through fifthexamples, the method further comprises adjusting temporal coherence ofthe single plurality of image frames based on the motion vectors. In aseventh example of the method optionally including one or more of thefirst through sixth examples, the method further comprises adjustingmotion blur of the single plurality of image frames based on the motionvectors. In an eighth example of the method optionally including one ormore of the first through seventh examples, the method further comprisesinterpolating image frames between adjacent image frames of eachplurality of image frames based on the motion vectors and the adjacentimage frames, and combining the plurality of image frames from eachimage frame source into the single plurality of image frames based onthe interpolated image frames and the motion vectors. In a ninth exampleof the method optionally including one or more of the first througheighth examples, a shutter angle for each image frame source isadjustable to control motion blur of the single plurality of imageframes.

In another representation, a method comprises acquiring image framesfrom two or more image frame sources with a first frame rate andstaggered acquisition times, arranging the image frames from the two ormore image frame sources into a single set of image frames according toorder of acquisition, calculating motion vectors for the image framesfrom the two or more image frame sources, applying motion correction tothe single set of image frames based on the motion vectors to generatefinal image frames, and outputting the final image frames as a videowith a second frame rate higher than the first frame rate.

In a first example of the method, calculating the motion vectors for theimage frames from the two or more image frame sources comprisescalculating a first set of motion vectors between adjacent image framesin the single set of image frames, and applying the motion correction tothe single set of image frames based on the motion vectors comprisesapplying the motion correction to the single set of image frames basedon the first set of motion vectors to correct for potential image shiftsbetween image frame sources. In a second example of the methodoptionally including the first example, calculating the motion vectorsfor the image frames from the two or more image frame sources furthercomprises calculating a second set of motion vectors between adjacentimage frames from each image frame source, and applying the motioncorrection to the single set of image frames based on the motion vectorsfurther comprises interpolating image frames at phases between phases ofthe adjacent image frames from each image frame source based on theadjacent image frames and respective motion vectors of the second set ofmotion vectors, and performing image fusion of the interpolated imageframes with image frames of the single set of image frames based on thefirst set of motion vectors. In a third example of the method optionallyincluding one or more of the first and second examples, the two or moreimage frame sources comprises five image frame sources, the first framerate comprises 24 frames per second, and the second frame rate comprises120 frames per second. In a fourth example of the method optionallyincluding one or more of the first through third examples, the staggeredacquisition times are adjusted based on a desired look and feel of thevideo.

In yet another representation, a system comprises an image processorconfigured with instructions in non-transitory memory that when executedcause the image processor to: acquire, via a plurality of image framesources configured with different acquisition settings, a plurality ofimage frames from each image frame source, each plurality of imageframes acquired at a first frame rate and with a different initialacquisition time than other image frame sources; combine the pluralityof image frames from each image frame source into a single plurality ofimage frames with a second frame rate higher than the first frame rate;and output the single plurality of image frames as a video with thesecond frame rate.

In a first example of the system, the image processor is furtherconfigured with instructions in the non-transitory memory that whenexecuted cause the image processor to calculate motion vectors betweenimage frames in each plurality of image frames and between image framesfrom different image frame sources. In a second example of the systemoptionally including the first example, the image processor is furtherconfigured with instructions in the non-transitory memory that whenexecuted cause the image processor to adjust temporal coherence of thesingle plurality of image frames based on the motion vectors. In a thirdexample of the system optionally including one or more of the first andsecond examples, the image processor is further configured withinstructions in the non-transitory memory that when executed cause theimage processor to adjust motion blur of the single plurality of imageframes based on the motion vectors. In a fourth example of the systemoptionally including one or more of the first through third examples,the image processor is further configured with instructions in thenon-transitory memory that when executed cause the image processor tointerpolate image frames between adjacent image frames of each pluralityof image frames based on the motion vectors and the adjacent imageframes, and combine the plurality of image frames from each image framesource into the single plurality of image frames based on theinterpolated image frames and the motion vectors.

As used herein, an element or step recited in the singular and proceededwith the word “a” or “an” should be understood as not excluding pluralof said elements or steps, unless such exclusion is explicitly stated.Furthermore, references to “one embodiment” of the present invention arenot intended to be interpreted as excluding the existence of additionalembodiments that also incorporate the recited features. Moreover, unlessexplicitly stated to the contrary, embodiments “comprising,”“including,” or “having” an element or a plurality of elements having aparticular property may include additional such elements not having thatproperty. The terms “including” and “in which” are used as theplain-language equivalents of the respective terms “comprising” and“wherein.” Moreover, the terms “first,” “second,” and “third,” etc. areused merely as labels, and are not intended to impose numericalrequirements or a particular positional order on their objects.

This written description uses examples to disclose the invention,including the best mode, and also to enable a person of ordinary skillin the relevant art to practice the invention, including making andusing any devices or systems and performing any incorporated methods.The patentable scope of the invention is defined by the claims, and mayinclude other examples that occur to those of ordinary skill in the art.Such other examples are intended to be within the scope of the claims ifthey have structural elements that do not differ from the literallanguage of the claims, or if they include equivalent structuralelements with insubstantial differences from the literal languages ofthe claims.

1. A method, comprising: acquiring image frames with a plurality ofimage frame sources configured with different acquisition settings;processing the image frames based on the different acquisition settingsto generate at least one final image frame; and outputting the at leastone final image frame.
 2. The method of claim 1, wherein acquiring theimage frames with the plurality of image frame sources configured withdifferent acquisition settings comprises acquiring a first set of imageframes forming a first video with a first image frame source and asecond set of image frames forming a second video with a second imageframe source, wherein the first set of image frames are acquired with alow resolution and a high frame rate, and the second set of image framesare acquired with a high resolution and a low frame rate.
 3. The methodof claim 2, further comprising aligning at least one image frame fromthe first set of image frames with an image frame from the second set ofimage frames.
 4. The method of claim 2, wherein processing the imageframes based on the different acquisition settings to generate the atleast one final image frame comprises estimating motion in the first setof image frames.
 5. The method of claim 4, wherein processing the imageframes based on the different acquisition settings to generate the atleast one final image frame further comprises sharpening the second setof image frames with a deconvolution kernel determined from theestimated motion in the first set of image frames.
 6. The method ofclaim 4, wherein processing the image frames based on the differentacquisition settings to generate the at least one final image framefurther comprises interpolating high-resolution image frames between theimage frames of the second set of image frames based on the estimatedmotion to generate a set of final image frames forming a final video. 7.The method of claim 1, wherein acquiring the image frames with theplurality of image frame sources comprises acquiring a first set ofimage frames with a first image frame source, each image frame of thefirst set of image frames acquired with a different exposure setting,and acquiring a second set of image frames with a second image framesource, wherein the second set of image frames are acquired withconsistent exposure settings.
 8. The method of claim 7, whereinprocessing the image frames based on the different acquisition settingsto generate at least one final image frame comprises estimating motionin the second set of image frames, and interpolating the first set ofimage frames based on the estimated motion in the second set of imageframes to generate the at least one final image frame.
 9. The method ofclaim 1, further comprising arranging the image frames into a single setof image frames according to order of acquisition of the image frames,each image frame source configured with an acquisition frame rate,wherein the different acquisition settings comprises different initialstart times for each image frame source, and wherein intervals betweenthe initial start times for each image frame source are adjustable tocontrol an appearance of a video formed from the single set of imageframes.
 10. The method of claim 9, further comprising estimating motionbetween image frames in the single set of image frames, and applyingmotion compensation to the image frames based on the estimated motion togenerate a motion-corrected set of image frames with a frame rate higherthan the acquisition frame rate, wherein the at least one final imageframe comprises the motion-corrected set of image frames.
 11. A method,comprising: acquiring a first set of image frames with a first imageframe source; acquiring a second set of image frames with a second imageframe source, the first set of image frames acquired in parallel withthe second set of image frames; estimating motion in the second set ofimage frames; generating at least one corrected image frame from thefirst set of image frames and the estimated motion; and outputting theat least one corrected image frame to a display.
 12. The method of claim11, wherein the first image frame source acquires the first set of imageframes with a high resolution and a low frame rate, and wherein thesecond image frame source acquires the second set of image frames with alow resolution and a high frame rate.
 13. The method of claim 12,further comprising filtering the first set of image frames with adeconvolution kernel determined from the estimated motion to adjust blurin the first set of image frames.
 14. The method of claim 11, whereinthe first image frame source acquires the first set of image frames withalternating exposure settings, and wherein the second image frame sourceacquires the second set of image frames with a single set of exposuresettings.
 15. The method of claim 11, further comprising aligning atleast one image frame from the first set of image frames with acorresponding image frame from the second set of image frames.
 16. Asystem, comprising: an image processor configured with instructions innon-transitory memory that when executed cause the image processor to:acquire, via a plurality of image frame sources configured withdifferent acquisition settings, a plurality of image frames; process theplurality of image frames based on the different acquisition settings togenerate at least one final image frame; and output the at least onefinal image frame to a display.
 17. The system of claim 16, wherein, toacquire the image frames with the plurality of image frame sourcesconfigured with different acquisition settings, the image processor isconfigured with instructions that when executed cause the imageprocessor to acquire a first set of image frames with a first imageframe source and a second set of image frames with a second image framesource, wherein the first set of image frames are acquired with a lowresolution and a high frame rate, and the second set of image frames areacquired with a high resolution and a low frame rate.
 18. The system ofclaim 17, wherein, to process the image frames based on the differentacquisition settings to generate the at least one final image frame, theimage processor is further configured with instructions that whenexecuted cause the image processor to estimate motion in the first setof image frames, and interpolate high-resolution image frames betweenthe image frames of the second set of image frames based on theestimated motion to generate a set of final image frames.
 19. The systemof claim 16, wherein, to acquire the image frames with the plurality ofimage frame sources, the image processor is configured with instructionsthat when executed cause the image processor to acquire a first set ofimage frames with a first image frame source, each image frame of thefirst set of image frames acquired with a different exposure setting,and acquire a second set of image frames with a second image framesource, wherein the second set of image frames are acquired withconsistent exposure settings.
 20. The system of claim 19, wherein, toprocess the plurality of image frames based on the different acquisitionsettings to generate at least one final image frame, the image processoris further configured with instructions that when executed cause theimage processor to estimate motion in the second set of image frames,and interpolate the first set of image frames based on the estimatedmotion in the second set of image frames to generate the at least onefinal image frame.